test: setup E2E testing via Playwright and implement some tests #88

patricktree · 2025-04-06T20:25:12Z

Prerequisites checklist

I have read the contributing guidelines.

What is the purpose of this pull request?

Introduces End-to-End (E2E) testing using Playwright to prevent regressions and increase confidence during development and refactoring.

What changes did you make? (Give an overview)

Set up Playwright for E2E testing.
- Configured GitHub Actions CI to run E2E tests.
Implemented initial E2E test scenarios.
Also made some components here and there more accessible - which is useful for locating things in Playwright.

Is there anything you'd like reviewers to focus on?

Notes on using Playwright compared to other options

vitest browser mode, downsides:
- it is experimental currently
- lacks documentation
- looks like it has no tools like test recorder, time travel debugging, etc.
Storybook Testing, downsides:
- visual regression testing is only possible with their SaaS product "Chromatic" (https://storybook.js.org/docs/writing-tests/visual-testing)
- I tried it out and and it just does not feel that mature and polished like Playwright (e.g. the locator API, expect APIs of Playwright are just great; Playwright produces a great report which is expecially useful in CI, ...)

Successful CI run: https://github.com/eslint/code-explorer/actions/runs/20301038365/

Note: The initial version of this PR created a Docker setup for consistent screenshot assertions, but that was too fragile - see discussion in this PR.
Archived notes:

Details

The new requirement, having Docker installed, is quite heavy... Thing is that otherwise, screenshots created by screenshot assertions ([`toHaveScreenshot`](https://playwright.dev/docs/api/class-pageassertions#page-assertions-to-have-screenshot-1)) are not consistent, leading to flaky behavior. As [noted by Playwright](https://playwright.dev/docs/test-snapshots#introduction):

Browser rendering can vary based on the host OS, version, settings, hardware, power source (battery vs. power adapter), headless mode, and other factors. For consistent screenshots, run tests in the same environment where the baseline screenshots were generated.

Alternatives to Docker considered:

not using screenshot assertions at all.
- Problematic because they are the only way to assert aesthetics, which is important for ESLint Code Explorer
- also, they are quite useful to get "much bang for the buck" as they cover much of a web application fast, compared to "fine-grained assertions"
implement some workflow using GitHub Actions and Pull Request comments to create baseline screenshots only in CI (like https://medium.com/@haleywardo/streamlining-playwright-visual-regression-testing-with-github-actions-e077fd33c27c)
- Problem is: how to develop E2E tests locally then?
- Also, while we could drop the Docker requirement then, we would need to add instructions to install the system dependencies required for Playwright (https://playwright.dev/docs/browsers#install-system-dependencies) - seems like adding Playwright to a project adds a requirement to the development machines of contributors, one way or the other

netlify · 2025-04-06T20:25:35Z

✅ Deploy Preview for eslint-code-explorer ready!

Name	Link
🔨 Latest commit	`d2729be`
🔍 Latest deploy log	https://app.netlify.com/projects/eslint-code-explorer/deploys/695ceb29e5d1300008502c8a
😎 Deploy Preview	https://deploy-preview-88--eslint-code-explorer.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

nzakas · 2025-04-08T19:05:48Z

Thanks for getting this started. I don't have time to dig in right now. I'm also not super familiar with this kind of setup, so could use some advice from the rest of the team. @eslint/eslint-team

package.json

e2e-tests/code-editing-and-ast-interaction.test.ts

Copilot

Copilot reviewed 51 out of 52 changed files in this pull request and generated no comments.

Files not reviewed (1)

package.json: Language not supported

Comments suppressed due to low confidence (2)

src/components/ui/popover.tsx:27

[nitpick] Verify that the use of role 'dialog' on PopoverContent is semantically correct for a popover component; if the component is used to display menus or tips rather than modal dialogs, a different role may be more appropriate.

role="dialog"

src/components/tree-entry.tsx:109

Ensure that the li element is nested within a ul or ol element to maintain proper HTML semantics.

<li className="flex items-center gap-3">

snitin315 · 2025-04-13T14:20:02Z

I'm familiar with such setup, will review this pr

nzakas · 2025-04-18T21:02:13Z

@snitin315 friendly ping to not forget about this

snitin315

@pkerschbaum Are we not running Docker in CI? If that's the case, it kind of defeats the purpose of using Docker, since the CI environment would always be inconsistent.

patricktree · 2025-04-21T16:57:13Z

@pkerschbaum Are we not running Docker in CI? If that's the case, it kind of defeats the purpose of using Docker, since the CI environment would always be inconsistent.

It is always - locally and in CI - running via Docker.
The only difference is some settings related to networking.

nzakas · 2025-04-22T14:47:02Z

@pkerschbaum can you take a look at the merge conflicts?

nzakas · 2025-04-22T15:15:21Z

I tried running this locally and got this error:

> [email protected] test:e2e    
> cross-env PW_TEST_CONNECT_WS_ENDPOINT=ws://127.0.0.1:3000/ playwright test      
[WebServer] Unable to find image 'mcr.microsoft.com/playwright:v1.51.1-noble' locally
[WebServer] v1.51.1-noble: 
[WebServer] Pulling from playwright      
[WebServer] 107a4fb0af38: Pulling fs layer
[WebServer] 8c401718c984: Pulling fs layer
[WebServer] 39224aba52ce: Pulling fs layer
[WebServer] 08d1cdd3adad: Pulling fs layer
[WebServer] 08d1cdd3adad: Waiting        
[WebServer] 39224aba52ce: 
[WebServer] Verifying Checksum           
[WebServer] 39224aba52ce: Download complete
[WebServer] 107a4fb0af38: 
[WebServer] Verifying Checksum           
[WebServer] 107a4fb0af38: Download complete
[WebServer] 8c401718c984: Download complete
[WebServer] 107a4fb0af38: Pull complete
[WebServer] 8c401718c984: 
[WebServer] Pull complete                
[WebServer] 39224aba52ce: 
[WebServer] Pull complete                

Error: Timed out waiting 60000ms from config.webServer.

To open last HTML report run:

  npx playwright show-report

I'm running Windows 11.

playwright.config.ts

patricktree · 2025-05-20T07:54:22Z

I tried running this locally and got this error:

> [email protected] test:e2e    
> cross-env PW_TEST_CONNECT_WS_ENDPOINT=ws://127.0.0.1:3000/ playwright test      
[WebServer] Unable to find image 'mcr.microsoft.com/playwright:v1.51.1-noble' locally
[WebServer] v1.51.1-noble: 
[WebServer] Pulling from playwright      
[WebServer] 107a4fb0af38: Pulling fs layer
[WebServer] 8c401718c984: Pulling fs layer
[WebServer] 39224aba52ce: Pulling fs layer
[WebServer] 08d1cdd3adad: Pulling fs layer
[WebServer] 08d1cdd3adad: Waiting        
[WebServer] 39224aba52ce: 
[WebServer] Verifying Checksum           
[WebServer] 39224aba52ce: Download complete
[WebServer] 107a4fb0af38: 
[WebServer] Verifying Checksum           
[WebServer] 107a4fb0af38: Download complete
[WebServer] 8c401718c984: Download complete
[WebServer] 107a4fb0af38: Pull complete
[WebServer] 8c401718c984: 
[WebServer] Pull complete                
[WebServer] 39224aba52ce: 
[WebServer] Pull complete                

Error: Timed out waiting 60000ms from config.webServer.

To open last HTML report run:

  npx playwright show-report

I'm running Windows 11.

can you plz try again?

I could reproduce the issue, sometimes the Docker pull takes a long time.
So, I changed the package.json script to do an explicit pull, which means the Playwright timeout does not apply anymore.

nzakas · 2025-05-20T14:33:39Z

No luck.

$ npm run test:e2e

> [email protected] test:e2e 
> docker pull mcr.microsoft.com/playwright:v1.51.1-noble && cross-env PW_TEST_CONNECT_WS_ENDPOINT=ws://127.0.0.1:36719/ playwright test

v1.51.1-noble: Pulling from playwright
107a4fb0af38: Pull complete           8c401718c984: Pull complete           39224aba52ce: Pull complete           08d1cdd3adad: Pull complete           Digest: sha256:146d046a8d79a1b3a87596c4457b0b1c47f811bf4fc2cc1b99e873ae7f1cbbbd
Status: Downloaded newer image for mcr.microsoft.com/playwright:v1.51.1-noble
mcr.microsoft.com/playwright:v1.51.1-noble

What's next:
    View a summary of image vulnerabilities and recommendations → docker scout quickview mcr.microsoft.com/playwri
[WebServer] Listening on ws://0.0.0.0:36719/

Error: Timed out waiting 30000ms from config.webServer.
                                      
To open last HTML report run:

  npx playwright show-report


nzaka@X1-Laptop MINGW64 ~/projects/eslint/code-explorer (pr/pkerschbaum/88-1)
$ npm run test:e2e

> [email protected] test:e2e 
> docker pull mcr.microsoft.com/playwright:v1.51.1-noble && cross-env PW_TEST_CONNECT_WS_ENDPOINT=ws://127.0.0.1:36719/ playwright test

v1.51.1-noble: Pulling from playwright
Digest: sha256:146d046a8d79a1b3a87596c4457b0b1c47f811bf4fc2cc1b99e873ae7f1cbbbd
Status: Image is up to date for mcr.microsoft.com/playwright:v1.51.1-noble  
mcr.microsoft.com/playwright:v1.51.1-noble

What's next:
    View a summary of image vulnerabilities and recommendations → docker scout quickview mcr.microsoft.com/playwri
[WebServer] Error: listen EADDRINUSE: address already in use 0.0.0.0:36719
[WebServer] npm notice                
[WebServer] npm notice New major version of npm available! 10.9.2 -> 11.4.0 
[WebServer] npm notice Changelog: https://github.com/npm/cli/releases/tag/v11.4.0
[WebServer] npm notice To update run: npm install -g [email protected]
[WebServer] npm notice

Error: Process from config.webServer was not able to start. Exit code: 1    
                                      
To open last HTML report run:

  npx playwright show-report

lumirlumir · 2025-06-26T06:25:57Z

Ping @pkerschbaum

patricktree · 2025-06-28T09:36:25Z

No luck.

$ npm run test:e2e

> [email protected] test:e2e 
> docker pull mcr.microsoft.com/playwright:v1.51.1-noble && cross-env PW_TEST_CONNECT_WS_ENDPOINT=ws://127.0.0.1:36719/ playwright test

v1.51.1-noble: Pulling from playwright
107a4fb0af38: Pull complete           8c401718c984: Pull complete           39224aba52ce: Pull complete           08d1cdd3adad: Pull complete           Digest: sha256:146d046a8d79a1b3a87596c4457b0b1c47f811bf4fc2cc1b99e873ae7f1cbbbd
Status: Downloaded newer image for mcr.microsoft.com/playwright:v1.51.1-noble
mcr.microsoft.com/playwright:v1.51.1-noble

What's next:
    View a summary of image vulnerabilities and recommendations → docker scout quickview mcr.microsoft.com/playwri
[WebServer] Listening on ws://0.0.0.0:36719/

Error: Timed out waiting 30000ms from config.webServer.
                                      
To open last HTML report run:

  npx playwright show-report


nzaka@X1-Laptop MINGW64 ~/projects/eslint/code-explorer (pr/pkerschbaum/88-1)
$ npm run test:e2e

> [email protected] test:e2e 
> docker pull mcr.microsoft.com/playwright:v1.51.1-noble && cross-env PW_TEST_CONNECT_WS_ENDPOINT=ws://127.0.0.1:36719/ playwright test

v1.51.1-noble: Pulling from playwright
Digest: sha256:146d046a8d79a1b3a87596c4457b0b1c47f811bf4fc2cc1b99e873ae7f1cbbbd
Status: Image is up to date for mcr.microsoft.com/playwright:v1.51.1-noble  
mcr.microsoft.com/playwright:v1.51.1-noble

What's next:
    View a summary of image vulnerabilities and recommendations → docker scout quickview mcr.microsoft.com/playwri
[WebServer] Error: listen EADDRINUSE: address already in use 0.0.0.0:36719
[WebServer] npm notice                
[WebServer] npm notice New major version of npm available! 10.9.2 -> 11.4.0 
[WebServer] npm notice Changelog: https://github.com/npm/cli/releases/tag/v11.4.0
[WebServer] npm notice To update run: npm install -g [email protected]
[WebServer] npm notice

Error: Process from config.webServer was not able to start. Exit code: 1    
                                      
To open last HTML report run:

  npx playwright show-report

I just noticed that the error is Error: listen EADDRINUSE: address already in use 0.0.0.0:36719 - can you try it again and if the error occurs again, look into Docker desktop if a container is running and if yes, stop+delete it and try again?

nzakas · 2025-07-08T15:45:19Z

Same result on a clean try:

$ npm run test:e2e

> [email protected] test:e2e
> docker pull mcr.microsoft.com/playwright:v1.51.1-noble && cross-env PW_TEST_CONNECT_WS_ENDPOINT=ws://127.0.0.1:36719/ playwright test

v1.51.1-noble: Pulling from playwright
Digest: sha256:146d046a8d79a1b3a87596c4457b0b1c47f811bf4fc2cc1b99e873ae7f1cbbbd
Status: Image is up to date for mcr.microsoft.com/playwright:v1.51.1-noble
mcr.microsoft.com/playwright:v1.51.1-noble

What's next:
    View a summary of image vulnerabilities and recommendations → docker scout quickview mcr.microsoft.com/playwright:v1.51.1-nob
[WebServer] Listening on ws://0.0.0.0:36719/

Error: Timed out waiting 30000ms from config.webServer.

To open last HTML report run:

  npx playwright show-report

patricktree · 2025-07-13T09:18:16Z

I assume you have Docker Desktop in use, so could you please try this: go to settings --> Resources --> Network --> make sure "Enable host networking" is checked.
Then, try it again.
Let's think then about how to proceed with this heavy-but-powerful E2E test setup :)

patricktree · 2025-10-11T08:56:32Z

I agree that an E2E testing setup which does not run easily on every development machine is bad as it makes it difficult / annoying to contribute to the project.
I wonder what's the best alternative:

Not doing screenshot assertions (would allow to remove the Docker things from the setup).
Switch to an entire other solution (e.g. Storybook got many updates in the recent past related to testing, maybe now it fits well the requirements of the Code Explorer project).

nzakas · 2025-10-24T14:46:46Z

I think whether or not we want screenshots is a fundamental question to this effort. So the question back to you: what is it you were hoping to catch with screenshots and is there another way to catch those problems?

Co-authored-by: Percy Ma <[email protected]>

…laywright to latest version

patricktree · 2025-12-17T11:26:35Z

I investigated this topic again:

I looked at other options for local visual testing (Loki, BackstopJS) but they also depend on Docker (for the same reason of consistent screenshots), so they would not solve the issue we have (brittle Docker setup).
Storybook solves this issue by uploading the entire application from the local device to their cloud (Chromatic), running tests there, reporting the results back - which feels very involved and I guess it would also make contributing to this project too difficult.

So I changed the E2E testing setup to a simple Playwright setup: no Docker anymore, also no screenshot assertions anymore.
That reduces the coverage of the tests, IMO it is still worth it as they validate that certain functionality works (typing in the code editor, expanding AST nodes, toggling theme, ...).

lumirlumir

Playwright is now working well in my Windows 11 development environment. If other team members have time to check, it would be nice to verify it across different environments.

I've also left a few additional comments throughout the codebase.

package.json

.github/workflows/ci.yml

playwright.config.ts

e2e-tests/code-editing-and-ast-interaction.test.ts

e2e-tests/options.test.ts

lumirlumir

LGTM. Thanks for all your hard work on this!

This looks like a good start toward adding E2E tests.

Would like @nzakas to verify before merging.

e2e-tests/options.test.ts

nzakas · 2026-01-06T21:03:34Z

When I run it locally, only Chromium succeeds. Everything else times out.

However, given that it appears to be running in CI and @lumirlumir indicates it has worked, I think that's good enough to plug it in to cover ourselves. Worst case scenario, we can encourage people to just use Chromium locally if they run into problems.

eslint-github-bot bot added the chore label Apr 6, 2025

patricktree marked this pull request as ready for review April 6, 2025 20:27

nzakas reviewed Apr 10, 2025

View reviewed changes

package.json Outdated Show resolved Hide resolved

e2e-tests/code-editing-and-ast-interaction.test.ts Show resolved Hide resolved

nzakas requested review from amareshsm and Copilot April 10, 2025 15:57

Copilot AI reviewed Apr 10, 2025

View reviewed changes

patricktree requested a review from nzakas April 13, 2025 14:11

snitin315 reviewed Apr 21, 2025

View reviewed changes

patricktree requested a review from snitin315 April 21, 2025 18:02

snitin315 reviewed Apr 29, 2025

View reviewed changes

playwright.config.ts Outdated Show resolved Hide resolved

patricktree force-pushed the test/setup-e2e-test branch from cd10bf7 to 2e1eeb9 Compare May 20, 2025 06:30

test: setup E2E tests

69dca82

patricktree force-pushed the test/setup-e2e-test branch from 2e1eeb9 to 69dca82 Compare May 20, 2025 06:59

patricktree added 4 commits May 20, 2025 09:36

ci: try to fix

0b349ae

debug

0240353

ci: try to fix

112e8a8

check process.env.CI for value "true"

0762f5b

patricktree and others added 5 commits December 17, 2025 10:52

Merge remote-tracking branch 'origin/main' into test/setup-e2e-test

a4a3b79

chore: update CI.yml action versions

c727b9b

Update playwright.config.ts

b9e5174

Co-authored-by: Percy Ma <[email protected]>

refactor: remove Docker setup, remove screenshot assertions, update p…

1a4b4df

…laywright to latest version

ci: fix

f198d04

patricktree requested review from kecrily and snitin315 December 17, 2025 11:26

lumirlumir self-requested a review January 2, 2026 11:27

lumirlumir requested changes Jan 2, 2026

View reviewed changes

patricktree added 7 commits January 4, 2026 21:56

Merge remote-tracking branch 'origin/main' into test/setup-e2e-test

0bf6453

test: rename e2e specs to .test

df0b8fc

test: cover HTML language option

71379e2

test: simplify Playwright config

a92bc90

chore: remove unused cross-env

a79a850

ci: bump upload-artifact to v6

ffab69f

cleanup

b1542cc

patricktree requested a review from lumirlumir January 4, 2026 21:23

nzakas added the contributor pool label Jan 5, 2026

lumirlumir previously approved these changes Jan 6, 2026

View reviewed changes

e2e-tests/options.test.ts Outdated Show resolved Hide resolved

lumirlumir moved this from Implementing to Second Review Needed in Triage Jan 6, 2026

test: include HTML in language list

d2729be

patricktree dismissed lumirlumir’s stale review via d2729be January 6, 2026 10:59

nzakas approved these changes Jan 6, 2026

View reviewed changes

nzakas merged commit 5ef61a7 into eslint:main Jan 6, 2026
9 checks passed

github-project-automation bot moved this from Second Review Needed to Complete in Triage Jan 6, 2026

patricktree deleted the test/setup-e2e-test branch January 9, 2026 09:24

Uh oh!

test: setup E2E testing via Playwright and implement some tests #88

test: setup E2E testing via Playwright and implement some tests #88

Uh oh!

Conversation

patricktree commented Apr 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Prerequisites checklist

What is the purpose of this pull request?

What changes did you make? (Give an overview)

Is there anything you'd like reviewers to focus on?

Uh oh!

netlify bot commented Apr 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for eslint-code-explorer ready!

Uh oh!

nzakas commented Apr 8, 2025

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

snitin315 commented Apr 13, 2025

Uh oh!

nzakas commented Apr 18, 2025

Uh oh!

snitin315 left a comment

Choose a reason for hiding this comment

Uh oh!

patricktree commented Apr 21, 2025

Uh oh!

nzakas commented Apr 22, 2025

Uh oh!

nzakas commented Apr 22, 2025

Uh oh!

Uh oh!

patricktree commented May 20, 2025

Uh oh!

nzakas commented May 20, 2025

Uh oh!

lumirlumir commented Jun 26, 2025

Uh oh!

patricktree commented Jun 28, 2025

Uh oh!

nzakas commented Jul 8, 2025

Uh oh!

patricktree commented Jul 13, 2025

Uh oh!

patricktree commented Oct 11, 2025

Uh oh!

nzakas commented Oct 24, 2025

Uh oh!

patricktree commented Dec 17, 2025

Uh oh!

lumirlumir left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lumirlumir left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nzakas commented Jan 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

patricktree commented Apr 6, 2025 •

edited

Loading

netlify bot commented Apr 6, 2025 •

edited

Loading